<!DOCTYPE html>
<html>
<head>
  <meta charset="utf-8">
  <meta name="description"
        content="Tutorials of ACL 2024: Watermarking for Large Language Model">
  <meta name="keywords" content="LLM, Watermark, ACL">
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <title>Watermarking for Large Language Model</title>

  <!-- Global site tag (gtag.js) - Google Analytics -->
  <script async src="https://www.googletagmanager.com/gtag/js?id=G-PYVRSFMDRL"></script>
  <script>
    window.dataLayer = window.dataLayer || [];

    function gtag() {
      dataLayer.push(arguments);
    }

    gtag('js', new Date());

    gtag('config', 'G-PYVRSFMDRL');
  </script>

  <link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Google+Sans|Noto+Sans|Castoro">
  <link rel="stylesheet" href="./static/css/bulma.min.css">
  <link rel="stylesheet" href="./static/css/bulma-carousel.min.css">
  <link rel="stylesheet" href="./static/css/bulma-slider.min.css">
  <link rel="stylesheet" href="./static/css/fontawesome.all.min.css">
  <link rel="stylesheet" href="https://cdn.jsdelivr.net/gh/jpswalsh/academicons@1/css/academicons.min.css">
  <link rel="stylesheet" href="./static/css/index.css">
  <link rel="icon" type="image/svg+xml" href="./static/images/favicon.svg">

  <script src="https://ajax.googleapis.com/ajax/libs/jquery/3.5.1/jquery.min.js"></script>
  <script defer src="./static/js/fontawesome.all.min.js"></script>
  <script src="./static/js/bulma-carousel.min.js"></script>
  <script src="./static/js/bulma-slider.min.js"></script>
  <script src="./static/js/index.js"></script>
</head>
<body>

  <nav class="navbar" role="navigation" aria-label="main navigation">
    <div class="navbar-brand">
      <a role="button" class="navbar-burger" aria-label="menu" aria-expanded="false">
        <span aria-hidden="true"></span>
        <span aria-hidden="true"></span>
        <span aria-hidden="true"></span>
      </a>
    </div>
    <div class="navbar-menu">
      <div class="navbar-start" style="flex-grow: 1; justify-content: center;">
        <a class="navbar-item" href="https://leililab.github.io/llm_watermark_tutorial/">
          <span class="icon">
            <i class="fas fa-home"></i>
          </span>
        </a>
        
        <!-- Additional Navbar Items with Text -->
        <a class="navbar-item" href="#related">
          Schedule
        </a>
        <a class="navbar-item" href="#BibTeX">
          BibTeX
        </a>
        </div>
      </div>
    </div>
  </nav>


<section class="hero">
  <div class="hero-body">
    <div class="container is-max-desktop">
      <div class="columns is-centered">
        <div class="column has-text-centered">
          <div class="hero-body" style="text-align: center;">
            <div style="display: flex; align-items: center; justify-content: center;">
              <!-- <img src="./static/images/acl.png" alt="Teaser Image" style="width: 30%; height: auto; margin: 0;"> -->
              
              <div style="margin-left: 20px; text-align: left;">
                <h1 class="title is-1 publication-title" style="margin-top: 0;"> 
                  <span style="font-family: 'Monoton', cursive; color: #4A90E2; font-size: 0.8em; text-shadow: 2px 2px 4px rgba(0, 0, 0, 0.2); letter-spacing: 2px; display: inline-block;">
                    Tutorials of ACL, ADL, NeurIPS 2024
                  </span>
                </h1>
              </div>
            </div>
            
            <h1 class="title is-1 publication-title" style="margin-top: 0;"> 
              Watermarking for Large Language Model
            </h1>
          </div>
        
          <div class="is-size-5 publication-authors" style="display: flex; justify-content: center; align-items: center; flex-wrap: wrap;">
            <span class="author-block" style="display: flex; align-items: center; margin-bottom: 10px; flex-wrap: wrap;">
                <div style="display: flex; align-items: center; margin-bottom: 10px; margin-left: 10px; margin-right: 10px;">
                    <img src="./static/images/xuandong.jpg" alt="Xuandong Zhao" style="width:100px; height:100px; border-radius:50%; object-fit:cover; margin-right:10px;">
                    <div>
                        <a href="https://xuandongzhao.github.io/">Xuandong Zhao</a>
                        <p style="margin: 0;">UC Berkeley</p>
                    </div>
                </div>
                <div style="display: flex; align-items: center; margin-bottom: 10px; margin-left: 10px; margin-right: 10px;">
                    <img src="./static/images/yuxiang.jpg" alt="Yu-Xiang Wang" style="width:100px; height:100px; border-radius:50%; object-fit:cover; margin-right:10px;">
                    <div>
                        <a href="https://cseweb.ucsd.edu/~yuxiangw/">Yu-Xiang Wang</a>
                        <p style="margin: 0;">UC San Diego</p>
                    </div>
                </div>
                <div style="display: flex; align-items: center; margin-left: 10px; margin-right: 10px;">
                    <img src="./static/images/leili.png" alt="Lei Li" style="width:100px; height:100px; border-radius:50%; object-fit:cover; margin-right:10px;">
                    <div>
                        <a href="https://www.cs.cmu.edu/~leili/">Lei Li</a>
                        <p style="margin: 0;">CMU</p>
                    </div>
                </div>
            </span>
        </div>
        
        <style>
            @media screen and (max-width: 768px) {
                .author-block {
                    flex-direction: column;
                    align-items: center;
                    text-align: center;
                }
        
                .author-block img {
                    margin-bottom: 10px;
                }
        
                .publication-authors {
                    flex-direction: column;
                }
            }
        </style>
        

          <!-- <div class="is-size-5">
            <!-- 📅 August 11, 14:00 - 17:30 📍 Lotus Suite 5-7.  -->
            <!-- We will take Q&A at <a href="https://app.sli.do/event/xjqaDYxzVuEotynDwSdZFz" class="link blue hover-orange" target="_blank">the Slido link</a>. -->
          <!-- </div> -->
          <div class="is-size-5" style="background: linear-gradient(45deg, #4a90e2, #72c6ef); color: #fff; padding: 10px 20px; border-radius: 10px; text-align: center; animation: pulse 1.5s infinite; display: inline-block;">
            <a href="https://app.sli.do/event/xjqaDYxzVuEotynDwSdZFz" style="color: #fff; text-decoration: none; font-weight: bold; padding: 5px 15px; transition: 0.3s; display: inline-block;" target="_blank" onmouseover="this.style.color='#72c6ef';" onmouseout="this.style.color='#fff';">
              Join us for the Q&A on Slido
            </a>
          </div>
          
          <style>
            @keyframes pulse {
              0%, 100% { transform: scale(1); opacity: 1; }
              50% { transform: scale(1.1); opacity: 0.8; }
            }
          </style>
        </div>
      </div>
    </div>
  </div>
</section>



<section class="section">
  <div class="container is-max-desktop">
    <!-- Abstract. -->
    <div class="columns is-centered has-text-centered">
      <div class="column is-four-fifths">
        <h2 class="title is-3">Abstract</h2>
        <div class="content has-text-justified">
          <p>
            Generative AI has significantly advanced, particularly in natural language processing, exemplified by models like ChatGPT, but these advancements have raised concerns about misuse, such as generating fake news or plagiarizing content. This tutorial explores text watermarking as a solution, embedding detectable patterns within AI-generated text to verify its origin. We will cover the evolution of text watermarking, its modern techniques, and challenges, along with model watermarking for copyright protection. Participants will gain a solid understanding of watermarking methods, their practical applications, and future research directions in this critical field.
          </p>
        </div>
      </div>
    </div>
    <!--/ Abstract. -->
  </div>
</section>


<section class="section" id="related">
  <div class="columns is-centered has-text-centered">
  <h2 class="title is-3">Schedule</h2>
  </div>
  <div class="container is-max-desktop">
    <!-- Concurrent Work. -->
    <div class="columns is-centered">
      <div class="column is-full-width">
        
        <h3 class="title is-4">Part I: Introduction [<a href="NeurIPS-tut-LLM-watermark-part-1.pdf">slides</a>]</h3>
        <div class="content has-text-justified">
          <p><strong>Presenter:</strong> Xuandong Zhao</p>
          <p>This section provides the background for the tutorial, presenting the challenges posed by machine-written text and the potential ethical issues arising from Large Language Models (LLMs). We will introduce two primary approaches to addressing these issues: post-hoc detection and watermarking methods.</p>
        </div>
  
        <h3 class="title is-4">Part II: Text Watermarking [<a href="NeurIPS-tut-LLM-watermark-part-2.pdf">slides</a>]</h3>
        <div class="content has-text-justified">
          <p><strong>Presenter:</strong> Xuandong Zhao, Yu-Xiang Wang</p>
          <p>We delve into the process and evolution of watermarking for natural languages. We further explore watermarking methods specifically designed for Large Language Models. We will also cover the theoretical analysis of each watermarking method.</p>
          <ul>
            <li>Early stages of text watermarking
            <li>Watermarking for Large Language Models:
              <ul>
                  <li>
                      KGW (Green-Red) Watermark: Kirchenbauer et al. (2023) 
                      <a href="https://arxiv.org/abs/2301.10226" target="_blank">
                          A Watermark for Large Language Models
                      </a>
                  </li>
                  <li>
                      Unigram (Green-Red) Watermark: Zhao et al. (2023) 
                      <a href="https://arxiv.org/abs/2306.17439" target="_blank">
                          Provable Robust Watermarking for AI-Generated Text
                      </a>
                  </li>
                  <li>
                      Gumbel Watermark: Aaronson (2023) 
                      <a href="https://www.youtube.com/watch?v=2Kx9jbSMZqA" target="_blank">
                          Watermarking of Large Language Models
                      </a>
                  </li>
                  <li>
                      Undetectable Watermark: Christ et al. (2023) 
                      <a href="https://arxiv.org/abs/2306.09194" target="_blank">
                          Undetectable Watermarks for Language Models
                      </a>
                  </li>
                  <li>
                      Distortion-free Watermark: Kuditipudi et al. (2023) 
                      <a href="https://arxiv.org/abs/2307.15593" target="_blank">
                          Robust Distortion-free Watermarks for Language Models
                      </a>
                  </li>
                  <li>
                    PF Watermark: Zhao et al. (2024) 
                    <a href="https://arxiv.org/abs/2402.05864" target="_blank">
                      Permute-and-Flip: An Optimally Robust and Watermarkable Decoder for LLMs
                    </a>
                  </li>
                  <li>
                    Unbiased Watermark: Hu et al. (2023) 
                    <a href="https://arxiv.org/abs/2310.10669" target="_blank">
                      Unbiased Watermark for Large Language Models
                    </a>
                  </li>
                  <li>
                    Mark My Words: Piet et al. (2023) 
                    <a href="https://arxiv.org/abs/2312.00273" target="_blank">
                      Mark My Words: Analyzing and Evaluating Language Model Watermarks
                    </a>
                  </li>
                  <li>
                    PRC Watermark: Christ and Gunn. (2024) 
                    <a href="https://arxiv.org/abs/2402.09370" target="_blank">
                      Pseudorandom Error-Correcting Codes
                    </a>
                  </li>
                  <li>Other methods...</li>
              </ul>
              <li>Trade-offs among watermarking methods:
                <ul>
                  <img src="./static/images/trade_off.png" style="width: 600px; height: auto; margin: 0 auto; display: block;">
                </ul>
              </ul>
            </li>
          </ul>
        </div>
  
        <h2 class="title is-3" style="text-align: center;">&mdash;&mdash;&mdash;&mdash;&mdash;&mdash;&mdash;&mdash;&mdash; Break (30 mins) &mdash;&mdash;&mdash;&mdash;&mdash;&mdash;&mdash;&mdash;&mdash;</h2>
  
        <h3 class="title is-4">Part III: Model Watermarking [<a href="ACL-tut-LLM-watermark-part-3.pdf">slides</a>]</h3>
        <div class="content has-text-justified">
          <p><strong>Presenter:</strong> Lei Li</p>
          <p>This part explores other related watermarking methods for AI models. We will discuss various methods used to safeguard intellectual property using watermarks.</p>
          <ul>
            <li>Copyright protection against model extraction attack:
                <ul>
                    <li>CATER: He et al. (2022) 
                        <a href="https://arxiv.org/abs/2209.08773" target="_blank">
                            CATER: Intellectual Property Protection on Text Generation APIs via Conditional Watermarks
                        </a>
                    </li>
                    <li>DRW: Zhao et al. (2022) 
                        <a href="https://arxiv.org/abs/2210.03312" target="_blank">
                            Distillation-Resistant Watermarking for Model Protection in NLP
                        </a>
                    </li>
                    <li>Ginsew: Zhao et al. (2023) 
                        <a href="https://arxiv.org/abs/2302.03162" target="_blank">
                            Protecting Language Generation Models via Invisible Watermarking
                        </a>
                    </li>
                    <li>Radioactivity: Sander et al. (2024) 
                        <a href="https://arxiv.org/pdf/2402.14904v1" target="_blank">
                            Watermarking Makes Language Models Radioactive
                        </a>
                    </li>
                </ul>
            </li>
            <li>Model detection against finetuning or pruning:
                <ul>
                    <li>DeepJudge: Chen et al. (2021)
                        <a href="https://arxiv.org/abs/2112.05588" target="_blank">
                            Copy, Right? A Testing Framework for Copyright Protection of Deep Learning Models
                        </a>
                    </li>
                    <li>Instructional Fingerprinting: Xu et al. (2024) 
                        <a href="https://arxiv.org/pdf/2401.12255" target="_blank">
                            Instructional Fingerprinting of Large Language Models
                        </a>
                    </li>
                </ul>
            </li>
        </ul>        
        </div>
  
        <h3 class="title is-4">Part IV: Post-Hoc Detection [<a href="ACL-tut-LLM-watermark-part-4.pdf">slides</a>]</h3>
        <div class="content has-text-justified">
          <p><strong>Presenter:</strong> Lei Li</p>
          <p>This part focuses on post-hoc detection methods, detailing the use of binary classifiers and statistical outlier detection techniques. We will discuss the theoretical and empirical limitations.</p>
        </div>
  
        <h3 class="title is-4">Part V: Conclusion and Future Directions [<a href="ACL-tut-LLM-watermark-part-5.pdf">slides</a>]</h3>
        <div class="content has-text-justified">
          <p>In the final section, we will provide a summary of the tutorial, discuss the best practices for implementing text watermarking, and explore potential future developments in the field of text watermarking.</p>
        </div>
  
        <h3 class="title is-4">Part VI: Q&A</h3>
      </div>
    </div>
    <!--/ Concurrent Work. -->
  </div>
</section>


<section class="section" id="BibTeX">
  <div class="container is-max-desktop content">
    <h2 class="title">BibTeX</h2>
    <pre><code>@article{zhao2024tutorials,
  author    = {Zhao, Xuandong and Wang, Yu-Xiang and Li, Lei},
  title     = {Watermarking for Large Language Model},
  journal   = {Tutorials of ACL},
  year      = {2024},
}</code></pre>
  </div>
</section>


<footer class="footer">
  <div class="container">
    <div class="columns is-centered">
      <div class="column is-8">
        <div class="content">
          <p>
            This website is licensed under a <a rel="license"
                                                href="http://creativecommons.org/licenses/by-sa/4.0/">Creative
            Commons Attribution-ShareAlike 4.0 International License</a>.
          </p>
          <p>
            This means you are free to borrow the <a
              href="https://github.com/nerfies/nerfies.github.io">source code</a> of this website,
            we just ask that you link back to this page in the footer.
            Please remember to remove the analytics code included in the header of the website which
            you do not want on your website.
          </p>
        </div>
      </div>
    </div>
  </div>
</footer>

</body>
</html>
